Continuity of the Value of Competitive Markov Decision Processes

A Markov Decision Process (MDP) is given by (i) a finite set of states S and an initial state s1 ¥ S, (ii) a finite set of actions A, (iii) a cost function c: S×AQ R, and (iv) a transition rule p: S×AQ D(S), where D(S) is the space of probability distributions over S. At every stage n ¥ N, where N is the set of positive integers, the process is in some state sn ¥ S. The decision maker chooses an action an ¥ A, and a new state sn+1 ¥ S is chosen according to p( · | sn, an). It is assumed that the decision maker remembers the sequence of states the process visited and his past actions. Denote by H=1n ¥ N (S×A)×S the set of all finite histories, where by convention, B=” for every finite set B and we identify ”×S with S. A plan of the decision maker is a function s which assigns to every finite